Exploring the MMPI-2 L Scale Cutoff In Police Selection

The present study investigated the use of the MMPI-2 L (Lie) scale as a criterion variable in the pre-employment selection of police officers. In a prior article, Weiss, Davis, Rostow, and Kinsman (2003) found that high L scale scores are associated with a number of performance problems in law enforcement officers. These investigators recommended a L scale cutoff of 8 when making hiring decisions. The present study sought to explore the usefulness of this cutoff by analyzing data from 4348 officers who had taken the MMPI-2 as a condition of pre-employment and then had follow up data from performance provided by their supervisors. The study found that officers with L scale raw scores of 8 or higher had significantly more performance problems than those who had scores of 7 or below. In addition, chi-square analyses were performed on individual performance criteria, and it was discovered that high L scale scorers engaged in a number of problem behaviors on the job, and were more likely to be terminated for cause. Implications of these findings for police psychological evaluations are discussed. EXPLORING THE MMPI-2 L SCALE 3 Exploring the MMPI-2 L Scale Cutoff in Police Selection Psychological evaluations as a prerequisite for employment in the field of law enforcement have become commonplace in recent years. These evaluations are performed in order to eliminate applicants who exhibit psychopathology or problematic personality characteristics that could interfere with their performance as a law enforcement officer. Due to the Americans With Disabilities Act, which stipulates that tests of psychopathology are medical tests and can only be used after all other factors are considered, such evaluations are the final stage of the hiring process (Weiss, 2010). Within such evaluations, the MMPI-2 is an instrument that is familiar to nearly every psychologist working in this area. Over the years, it has become one of the most popular instruments used by psychologist for pre-employment psychological evaluations of law enforcement personnel (Weiss & Weiss, 2010). Perhaps the first formal investigation of the power of the MMPI to assist with candidate selection was that performed by King, Norrell, and Erlandson (1959) who found that the MMPI-2 could be used to predict police academy grades, and that the Basic Scales were useful at identifying problem officers. Since that time, multiple articles have been published which have established the utility of the MMPI and MMPI-2 for predicting law enforcement officer performance. At one time or another, various studies have shown that high scores on Scales 1,2,4,6,7,8,9, and 0 are indicative of future performance problems as a law enforcement officer (Weiss & Weiss, 2010). Interestingly, Sellbom, Fischler, and Ben-Porath (2007) found that lower scores on Scale 3 were associated with performance problems. Brewster and Stoloff (1999) suggest that a law enforcement applicant with any Basic Scale score above T = 65 may be problematic EXPLORING THE MMPI-2 L SCALE 4 based on their study, as well as previous ones, and can be potentially removed from the applicant pool. In addition, a few of the Supplementary Scales, including O-H (Weiss, Johnson, Serafino, & Serafino, 2001) and MAC-R (Sellbom, Fischler, & Ben-Porath, 2007) have also been associated with poor performance of law enforcement officers. As mentioned, law enforcement applicants go through a high degree of prescreening prior to the psychological evaluation. Such individuals generally receive extensive background checks, interviews, fitness and drug tests, and even polygraph tests (Weiss, Weiss, & Gacono, 2008) prior to a conditional offer of hire. This being the case , individuals with obvious psychological adjustment problems are usually eliminated from the applicant pool long before they ever get the chance to see the police psychologist. This procedural practice results in a situation where the Basic Scale scores on the MMPI2 are nearly always in the normal range for remaining candidates, and in fact are often below the established norms (Hitchcock, Weiss, Weiss, Rostow, & Davis, 2010). Elevated scores on the Basic Scales are thus quite rare for these candidates. For this reason, investigators have for some time been exploring other MMPI-2 scales that are less dependent on constructs related to psychopathology, especially the validity scales. One validity scale that has fairly strong support for use as a predictor of future law enforcement performance is the L (Lie) scale. The L Scale was previously investigated as a law enforcement selection tool by Weiss, Davis, Rostow, and Kinsman (2003). Their study is based on earlier work by Boes, Chandler, and Timm (1997) and Herndon (1998) which suggested that the L scale may be useful in police selection because it is a more subtle variable. Individuals who are high scorers on the L scale are attempting to present themselves as exceptionally virtuous, and often may be minimizing EXPLORING THE MMPI-2 L SCALE 5 distress and psychopathology. Such responding is more common in law enforcement populations than reports of psychopathology and distress on the Clinical Scales. In the Weiss et al. (2003) study, pre-employment data from 1347 law enforcement officers was correlated with a series of performance criterion variables, which were obtained through follow-up one year after hire. Specifically, L-scale scores were found to be significantly associated with termination for cause, knowledge mistakes, failure to complete requirements for conditional hire, insubordination, and resignation/no longer employed by the department. Echoing Graham’s (2006) insights concerning high L-scale scorers, Weiss et al. (2003) point out that such individuals may not make good law enforcement personnel. These individuals claim to be excessively virtuous and deny any problems. They do not admit to even minor shortcomings, and essentially, are not being truthful in responding to the MMPI-2 items. These characteristics suggest that either they are attempting to be deceptive and hide negative personality characteristics (such as psychopathology), or they may simply be dishonest people. In addition, Weiss et al. (2003) mention that high L-scale scorers may also have a poor tolerance for stress, show little awareness of the consequences of their behavior, and also are unoriginal and inflexible in their thinking and problem solving. These characteristics would not be desirable in a high-risk profession such as policing, in which officers must follow orders but at the same time make quick, effective decisions in difficult situations. As a result of these findings, Weiss et al. (2003) recommend using a L scale raw score cutoff of 8 (T = 70 for males and T = 71 for females) for pre-employment hiring purposes. They state that individuals with scores of 8 or higher should be scrutinized carefully. As noted by Weiss, Weiss, Cain, and Manley (2009), such a cutoff makes EXPLORING THE MMPI-2 L SCALE 6 sense because the pre-employment situation pulls for slightly (but not dramatically) elevated L-scale scores as individuals tend to want to make a good impression. Such a cutoff allows for slight elevations but calls for careful scrutiny or even removal from the applicant pool when individuals obtain more dramatically elevated L-scale scores. The purpose of the present study was to test this proposed cutoff with a significantly larger sample of N = 4348 participants than was used in the Weiss et al. (2003) study (N = 1347). Further, whereas Weiss et al. (2003) utilized simple bivariate correlation to explore the relationship between L-scale scores and police performance, the present investigation compares job performance of candidates above and below cutoffs with a wide array of analytic procedures. The dependent variable was a measure of overall police officer performance previously used in a study by Weiss, Hitchcock, Weiss, Rostow, and Davis (2008). It was hypothesized that officers with an L raw score of 8 or higher would show significantly poorer job performance overall than those with an L raw of 7 or lower. In addition, nonparametric tests were performed on individual performance criteria (all follow-up questions were in a yes/no format) to shed light on the specific job-related problems of high scorers. In addition, the investigators were interested in comparing this suggested cutoff of 8 with other cutoffs. As stated earlier, Weiss et al. (2009) showed that the law enforcement pre-employment scenario appears to pull for moderate elevations (approximately T = 65, or an L raw of 7) in most subjects. Therefore, it is possible that a cutoff of 8 may be slightly low. In addition, in writing about the original MMPI, Dahlstrom, Welsh, and Dahlstrom (1972) stated that the point at which clinical scale scores become substantially suppressed due to dishonest responding is at an L raw of 9. EXPLORING THE MMPI-2 L SCALE 7 However, the usual cutoff for an elevated L score which would invalidate an MMPI profile is a raw score of 7, which corresponds to a T-score of 65 for males and 66 for females. Therefore, exploratory analyses were conducted using cutoffs of 7 and 9 as well, in order to determine the most appropriate cut point for the L scale in performing police selection work with the MMPI-2. While the present researchers would have liked to have tried higher cutoffs, no follow-up data for officers with L scale scores of 11 or higher were available, as a raw score of 11 to 15 constituted an automatic “failure” for the police departments involved, and so no individuals in the present data set with scores in these ranges actually worked as law enforcement personnel. Method Participants The participants in the present study were 4348 individuals who had been given conditional offers of employment as a law enforcement officer pending a psychological evaluation. They were recruited from multiple police departments in a variety of southern states. 3701 of the officers were male, and 647 were female. Instruments and Procedure All of the officers were administered the MMPI-2 as part of a pre-employment psychological evaluation conducted by one of the last two authors. In order to test the cutoff score proposed by Weiss et al. (2003) their MMPI-2 profiles were scored for the L scale and then were divided into two groups. The high-L group consisted of 712 officers with raw scores on the L-scale that were 8 or higher. The low-L group contained the remaining 3636 individuals who had raw scores on the L-scale of 7 or less. Each participant had follow-up data which was obtained by the investigators post-employment. EXPLORING THE MMPI-2 L SCALE 8 The length of time on the police force prior to obtaining the follow-up data was variable, but the average was 1.5 years. These follow-up data were in the form of 32 yes-no questions which asked about potential performance problems. Each individual was given one “point” for an indicator of problem performance. The totals were summed across all 32 categories to give an indicator of problem performance. The range for the current sample was from 0 to 21 on the total performance measure . This total performance index was used as the main dependent variable in the present study. The performance criteria used for the present study can be found in Appendix A. Results A final sample of N = 4348 officers were available for analysis. The number of job-related problems comprising the police performance measure ranged from 0 to 21 with M = 2.22. Over 90% of the performance scores (# problems identified) were below 6. Because the overall police performance measure was somewhat positively skewed in this way, a square root transformation was performed on these data rendering them normally distributed. Comparisons of police performance on this overall measure above and below designated cutoffs were thus made on both data in their original form and after transformation. MMPI-2 L Scale Cutoffs and Police Performance The mean number of performance problems was compared above and below L Scale cutoffs of 8 and 7. As seen in Table 1, officers with scores at or above 8 were judged by supervisors to have had significantly more work-related problems (M = 2.39) than those with scores below this cutoff (M = 2.18), t (4346) = -2.17; p = . 030. The effect was slightly stronger on the transformed version of the performance data, t (4346) EXPLORING THE MMPI-2 L SCALE 9 = -2.26, p = .024. A similar pattern of results was obtained using an L Scale cutoff of (7). Here again, those scoring at or above the cutoff were reported to have had significantly more problems at work (M = 2.41) than those scoring below (M = 2.15), t (4346) = -3.19, p = .001. With the transformed data, this latter effect is even more pronounced (Cohen’s d = 0.14), but still small in overall magnitude (see Table 1). Data were also analyzed for a cutoff score of 9. While results were in the same direction as the other cutoffs, they were only marginally significant with the transformed data, t (4346) = -1.916, p = .055. Subsequent chi-square analyses, therefore, were only performed using raw score cutoffs of 7 and 8 which yielded more robust effects. MMPI L Scale Cutoffs and Specific Indicators of Police Performance To aid in our understanding of the job-related problems of officers scoring at various levels on the L Scale, individual items in the police performance questionnaire were crosstabulated with L Scale categories (above/below cutoffs of 8 & 7). All 32 performance criteria were included to determine what kinds of performance problems are associated with high L scale scores. In Tables 2 & 3 the results of chi-square and logistic regression analyses describe this association between L Scale cutoffs and specific jobrelated problems identified by supervisors. Though these analyses are somewhat redundant, they produce different statistics (percentages vs. Odds ratios) that allow us to communicate results in a more comprehensive fashion. The final, “adjusted” analyses allow us to determine how robust the bivariate results turn out to be. Results presented in the tables were restricted to those yielding significant chi-squares. Cutoff (8). L Scale scores of 8 – 10 were significantly associated with a variety of six specific performance problems including discharge of a weapon in the line of duty, EXPLORING THE MMPI-2 L SCALE 10 citizen complaints of unprofessional conduct, resignation for personal reasons, termination for cause, insubordination, and excessive absenteeism (see Table 2). Significant Odds Ratios indicate that high scorers (8-10) were 38%, 28%, and 34% more likely, respectively, than low scorers (0-7) to have citizen complaints against them, to voluntarily resign, or to be terminated for cause. More dramatically, high scorers were more than twice as likely to discharge a weapon in the line of duty (OR = 2.21), be guilty of insubordination (OR = 2.33) and be absent from work to an excessive degree (OR = 2.24). With the exception of termination for cause, these effects remained even after controlling for gender, ethnicity/race, and marital status. Cutoff (7). A similar, but not identical, pattern of results was obtained when the L Scale cutoff was lowered from 8 to 7. As seen in Table 3, significant associations were found between L Scale scores (0-6 vs. 7-10) and seven specific performance problems. As with a cutoff of 8, high scorers, this time defined as having scores between 7-10, were 79% more likely than low scorers (0-6) to discharge a weapon in the line of duty, 58% more likely to have citizen complaints of unprofessional conduct, and 37% more likely to resign for personal reasons. While termination for cause, insubordination, and excessive absenteeism were not significantly associated with this cutoff, other performance problems were. Officers scoring between 7-10 on the L Scale were 34% more likely than low scorers (0-6) to have engaged in undesirable off-duty conduct, 78% more likely to have complaints brought against them for excessive use of force, and 25% more likely to have been suspended or have written reprimands on their record. These high scorers were also more than twice as likely (OR = 2.11) to have been judged guilty of racially EXPLORING THE MMPI-2 L SCALE 11 offensive conduct. All of these effects associated with this cutoff were robust insofar as they remained significant after controlling for gender, ethnicity/race, and marital status. Discussion The primary hypothesis of the present study, that individuals with a raw score on the L scale of the MMPI-2 of 8 or higher would show significantly more performance problems after hire as police officers was supported. The 712 officers in the present sample who had L scale scores between 8 and 10 overall exhibited significantly more performance problems, as measured by the 32 performance criteria, than the officers who had scores between 0 and 7. In addition, exploratory analyses also discovered that Individuals with L scale scores of 7 or higher also exhibited more performance problems than those with scores in the 0 to 6 range. The overall results of the present study, therefore, confirm the conclusions of the Weiss et al. (2003) study. The present study also demonstrates that the L scale is a useful tool in the pre-employment screening of police officers, because it can to a degree discriminate between who will turn out to be a good or poor officer. Moreover, the present study also extended the findings of the Weiss et al. (2003) study by describing some characteristics of high-L scorers. Perhaps the most important finding is that individuals with raw scores of 8 or higher are more likely to be terminated for cause. As with any form of employment, police departments desire to eliminate potential employees at high risk for termination from the applicant pool. This is particularly crucial in law enforcement work, however, due to the high-risk nature of policing and the high expense and time commitment of hiring and training law enforcement personnel. In the present EXPLORING THE MMPI-2 L SCALE 12 sample, perhaps none of the variables explored better illustrates the problems associated with a high score on the L scale. In addition, several other blatant performance problems are associated with a raw score of 8-10. These include insubordination, excessive absenteeism, and citizen complaints about unprofessional conduct. All three of these performance problems, associated with high scores, represent behaviors that are clearly undesirable, particularly in the law enforcement environment in which organization, responsibility, and public respect are of the utmost importance. While the other two problems, resignation for personal reasons and discharging a weapon in the line of duty, do not appear at first glance to be problems as serious as the other three, these represent potential performance problems. While personal reasons may be quite broad, they may also represent occurrences or circumstances that negatively affect performance, and law enforcement organizations generally wish to hire individuals who stay with the job. In addition, discharging a weapon in the line of duty, despite what is often seen on television shows, is a very unusual occurrence in actual practice in law enforcement. The use of force of any kind is a rare event, and the use of firearms even less so (Lersch, Bazley, Mieczkowski, & Childs, 2008). Discharging a weapon in the line of duty is therefore potentially (but admittedly not necessarily) representative of impulse control or judgment problems. Using a cutoff of 7 actually produced a slightly stronger effect size in the overall analysis. Several performance problems which were not evident using the cutoff of 8 were more strongly associated with the 7-10 group than the 0-6 group. It would appear at first glance that 7 is a better cutoff, however, the present authors urge some caution in EXPLORING THE MMPI-2 L SCALE 13 making such decisions. First of all, termination for cause was not more likely using a cutoff of 7, but it was a more likely occurrence using a cutoff of 8. Termination for cause, in making hiring decisions, is probably an ultimate “gold standard” and is arguably the most important criterion variable investigated. In addition, Weiss et al. (2009) make the point that slightly elevated L scale scores (T = 65; roughly equivalent to a raw score of 7) are to be expected in the pre-employment scenario. Given the small but significant effect sizes, there is a potential confound between those individuals who are essentially being dishonest and those who are producing slightly elevated scores due to the demand characteristics of the situation. Therefore, 8 may be a superior cutoff for these reasons—it predicts termination for cause, and probably better identifies dishonest individuals with problem personality characteristics. The finding that a raw score of 9 or 10 was not predictive of future performance as a law enforcement officer, while exploratory, was interesting given that the effect was significant using a cutoff of 8. However, a raw score of 9 is equivalent to a T score of 78 and is therefore an unusual score, nearly three standard deviations above the mean. As a result, the high L sample was reduced from 712 to 375, or nearly halved. The resulting change in numbers resulted in a lack of power and, given the constricted nature of police personality pre-employment data (see below) produced a non-significant result. This finding is therefore probably related to the smaller sample size. While the present study shows the utility of the cutoff of 8, its effect sizes are small. However, the results of the present study should not be dismissed simply because of its small effects. Large effect sizes are almost never found in research on preemployment psychological evaluations of police officers (Weiss & Weiss, 2011) because EXPLORING THE MMPI-2 L SCALE 14 of the nature of the samples obtained by researchers. As stated earlier, individuals with obvious psychopathology or personality problems are generally eliminated from the applicant pool before the psychological evaluation, which is performed as the final part of the hiring process and after a conditional offer of employment has been made. Therefore, this results in a truncation of range in scores on personality inventories oriented towards measuring psychopathology, such as the MMPI-2. In addition, such a screening process results in high quality control; the vast majority of officers hired perform well. Therefore, it becomes very difficult to find high scorers on the MMPI-2 scales, as well as problem performers, in samples of law enforcement officers with extensive job performance data. For these reasons, Hitchcock, O’Conner, & Weiss (2010) demonstrate that small but significant effects are generally quite meaningful in personality assessment research in law enforcement selection, and should not be ignored. True random sampling, which would produce normally distributed data sets, is impossible in this type of research due to the risk to the public if unqualified individuals were hired as police officers (Weiss & Weiss, 2011). However, by the same token, very small effects should be carefully considered in police research, due to the potential risk to the public welfare. Law enforcement officers have, under certain circumstances, the right to take life and liberty, and the potential costs to society of a hiring mistake probably outweigh the costs of a “false positive” result indicative of problem performance. These results also have implications for future research. One important future application would be to explore follow up data with a wider range of L scale scores. Unfortunately, as stated earlier, follow up data for individuals with raw scores higher than 10 were not available in the present study. In addition, while the results of the EXPLORING THE MMPI-2 L SCALE 15 present study appear fairly robust, performing additional research on data sets with large numbers of problem officers would also help to further extend the present findings. Overall, the results of the present study provide support for the use of the L scale in police selection, and these results help to provide guidelines for psychologists in making recommendations to police departments. EXPLORING THE MMPI-2 L SCALE16 ReferencesBoes, J.O, Chandler, C.J., & Timm, H.W. (1997). Police integrity: Use of personalitymeasures to identify corruption-prone officers. Monterey, CA: Defense PersonnelSecurity Research Center.Brewster, J., & Stoloff, M.L. (1999). Using the good cop/bad cop profile with the MMPI-2. Journal of Police and Criminal Psychology, 14 (2), , 29-34.Dahlstrom, W.G., Welsh, G.S., & Dahlstrom, L.E. (1972). An MMPI handbookvolume 1: Clinical interpretation. (Rev. Ed.) Minneapolis: University ofMinnesota Press.Graham, J.R. (2006). MMPI-2: Assessing personality and psychopathology (4 ed.). NewYork: Oxford University Press.Herndon, J. (1998, October). Correlates of the MMPI-2 L scale: Elevations in an LEOselection test battery. Paper presented at the 27 Annual Meeting of the Societyfor Police and Criminal Psychology, Portland, OR.Hitchcock, J.H., Weiss, P.A., Weiss, W.U., Rostow, C.D., & Davis, R.D. (2010). Thefuture of personality assessment research in police psychology: What’s next andwhat do we need? In P.A. Weiss (Ed.), Personality assessment in police psychology: A 21 century perspective (pp. 279-296). Springfield, IL: Charles CThomas.King, P., Norrell, G., & Erlandson, F.L. (1959). The prediction of academic success in a police administration curriculum. Educational and Psychological Measurement, 19, 649-651.Lersch, K.M., Bazley, T., Mieczkowski, T., & Childs, K. (2008). Police use of force and EXPLORING THE MMPI-2 L SCALE17 neighborhood characteristics: An examination of structural disadvantage, crime,and resistance. Policing & Society, 18, 282-300.Sellbom, M., Fischler, G.L., & Ben-Porath, Y.S. (2007). Identifying MMPI-2 predictorsof police officer integrity and misconduct. Criminal Justice and Behavior, 34,985-1004.Weiss, P.A., Hitchcock, J.H., Weiss, W.U., Rostow, C., & Davis, R. (2008). ThePersonality Assessment Inventory borderline, drug, and alcohol scales aspredictors of overall performance in police officers: A series of exploratoryanalyses. Policing and Society, 18, 301-310.Weiss, P.A., & Weiss, W.U. (2010). Using the MMPI-2 in police psychologicalassessment. In P.A. Weiss (Ed.), Personality assessment in police psychology: A21 century perspective (pp. 59-71). Springfield, IL: Charles C Thomas.Weiss, P.A., & Weiss, W.U. (2011). Criterion-related validity in police psychologicalevaluations. In J. Kitaeff (Ed.), Handbook of police psychology (pp. 125-133).New York: Routledge.Weiss, P.A., Weiss, W.U., & Gacono, C.R. (2008). The use of the Rorschach in police psychology: Some preliminary thoughts. In C.B. Gacono & F.B. Evans (Eds.),The handbook of forensic Rorschach assessment (pp.527-542). New York:Lawrence Erlbaum Associates. Weiss, W.U. (2010). Procedural considerations in security personnel selection. In P.A.Weiss (Ed.), Personality assessment in police psychology: A 21 centuryperspective (pp. 299-316). Springfield, IL: Charles C Thomas.Weiss, W.U., Davis, R., Rostow, C., & Kinsman, S. (2003). The MMPI-2 L scale as a EXPLORING THE MMPI-2 L SCALE18 tool in police selection. Journal of Police and Criminal Psychology, 18(1), 57-60.Weiss, W.U., Johnson, J., Serafino, G., & Serafino, A. (2001). A three-year follow up ofthe performance of a class of state police academy graduates using the MMPI-2.Journal of Police and Criminal Psychology, 16, 51-55.Weiss, W.U, Weiss, P.A., Cain, S., & Manley, B. (2009). Impression management inpolice officer candidacy on the MMPI-2. Journal of Police and CriminalPsychology, 24, 120-125. EXPLORING THE MMPI-2 L SCALE19 Appendix APerformance Questionnaire Items Completed by Departmental Supervisors1. Did this officer have any procedural mistakes?2. Did this officer have any conduct mistakes?3. Did this officer have any knowledge mistakes?4. Has this officer been involved in an on-duty or off duty at-fault motor vehicleaccident?5. Has this officer discharged his or her weapon in the line of duty?6. Has this officer demonstrated an inappropriate use of any weapon?7. Has this officer demonstrated undesirable off-duty conduct?8. Has this officer been responsible for damage or destruction of official property?9. Has this officer misused official vehicles?10. Has this officer received any citizen complaints regarding the excessive use offorce?11. Has this officer received any citizen complaints regarding unprofessionalconduct? 12. Has this officer received any suspensions or written reprimands by superiors? 13. Has this officer been sued in any lawsuit in connection with sustainedmisconduct?14. Has this officer been involved in family difficulty? 15. Has this officer been charged, arrested, or detained for any misdemeanor orfelony? EXPLORING THE MMPI-2 L SCALE20 16. Has this officer been accused in any way of inappropriate sexual behavior,harassment, conflicts, indiscretion, or offensive behavior?17. Has this officer been accused in any way of racially offensive conduct, behavior,verbalizations, or complaints?18. Has there been any hint, suggestion, or evidence that this officer uses alcohol orany other abuseable chemical more or problematically compared to other officers?19. Did this officer resign voluntarily for personal reasons?20. Did this officer resign voluntarily for non-police work?21. Did this officer resign voluntarily for other police work?22. Did this officer resign at the request of the department?23. Was this officer terminated for cause?24. Was this officer’s conditional offer of employment withdrawn?25. Did this officer fail to complete the requirements of conditional hire?26. Did this officer fail to comply with departmental regulations?27. Did this officer engage in insubordination?28. Did this officer receive an excessive number of citizen complaints?29. Did this officer engage in corruption or criminal conduct? 30. Did this officer engage in neglect of duty? 31. Did this officer have excessive absenteeism?32. Did this officer have other performance problems not listed above?